Skip to content

Reduce vmcnt for fa - #224

Open
Annieren wants to merge 1 commit into
gfx11from
annieren.fa-reduce-latency
Open

Annieren wants to merge 1 commit into
gfx11from
annieren.fa-reduce-latency

Conversation

@Annieren

Copy link
Copy Markdown

Overview

This pr fix these two fa issues during single token decoding:

  1. asm volatile in ggml_cuda_mad causes a scheduling barrier
  2. In fattn-vec.cuh, per-row work inside the K loop: a mask load from global memory, running-max update for every row, which forces a wait for all earlier loads, and the branches block scheduling.

Additional information

Requirements

  1. This fix is bit-identical to base in terms of perplexity.
  2. Performance improvement of this pr:
model depth baseline t/s PR t/s delta
Janus-Pro-1B 128 188.12 194.28 +3.3%
Janus-Pro-1B 4096 92.95 105.73 +13.7%
Llama-2-7B 128 50.09 50.43 +0.7%
Llama-2-7B 1920 37.79 40.19 +6.4%
Llama-2-7B 4096 28.87 31.82 +10.2%
Llama-3.1-8B-kvq8_0 4096 40.00 37.56 -6.1%
SmolLM2-1.7B 128 165.67 165.83 +0.1%
SmolLM2-1.7B 4096 90.20 96.36 +6.8%
Qwen3-4B 128 79.19 78.09 -1.4%
Qwen3-4B 4096 65.16 65.10 -0.1%

Llama-3.1-8B-kvq8_0's regression will be addressed separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant